Skip to content

HBASE-30352 Tolerate stale recovered.edits below durable seqid in split/merge (branch-2 backport) - #8699

Open
nirdosh0110 wants to merge 1 commit into
apache:branch-2from
nirdosh0110:HBASE-30352-branch-2
Open

nirdosh0110 wants to merge 1 commit into
apache:branch-2from
nirdosh0110:HBASE-30352-branch-2

Conversation

@nirdosh0110

Copy link
Copy Markdown
Contributor

Backport of #8599 to branch-2 (JIRA: HBASE-30352).

Problem

On split/merge, AssignmentManagerUtil.checkClosedRegion aborts the procedure whenever any recovered.edits file exists for a parent region, to prevent data loss. In corner cases a region can be left with recovered.edits whose max seqid is already covered by the region's durable (last-flushed) seqid — those edits are provably redundant, yet their mere presence wedges the split/merge into a stuck RIT that requires manual cleanup.

Change

checkClosedRegion now tolerates such stale files: before aborting, it consults the master's last-flushed seqid for the region and, only when that watermark is authoritative (> 0), drops recovered.edits files whose max seqid is <= durableSeqId. If any file's seqid is above the watermark (or the watermark is unknown), it still aborts as before — no behavior change for the genuinely-unsafe case.

  • Scans the region WAL dir, region dir, and the "wrong" WAL region dir via WALSplitUtil.getSplitEditFilesSorted.
  • Deletes only files strictly at/below durableSeqId; a failed delete or an unparseable/higher seqid falls back to the safe abort.

Test

TestAssignmentManagerUtil covers the new path (3 tests, all green locally).

Verification (local, JDK17, -Dhadoop.profile=3.0)

  • mvn -pl hbase-server -am clean test -Dtest=TestAssignmentManagerUtil → Tests run: 3, Failures: 0, Errors: 0
  • mvn -pl hbase-server -am spotless:check → BUILD SUCCESS

Cherry-pick was clean apart from one import block (branch-2 uses CommonFSUtils/FSUtils; the master-only FutureUtils import was dropped as it has no usage on branch-2).

@nirdosh0110 nirdosh0110 changed the title HBASE-30352 Tolerate stale recovered.edits below durable seqid in split/merge HBASE-30352 Tolerate stale recovered.edits below durable seqid in split/merge (branch-2 backport) Sep 24, 2026
@virajjasani

Copy link
Copy Markdown
Contributor

Awaiting full github workflow results before merging master and branch-2 PRs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants